Processing source video for real-time enhancement of a signal of interest

ABSTRACT

What is disclosed is a system and method for real-time enhancement of an identified time-series signal of interest in a video that has a similar spatial and temporal structure to a given reference signal, as determined by a measure of closeness. A closeness measure is computed for pixels of each image frame of each channel of a multi-channel video to identify a time-series signal of interest. The intensity of pixels associated with that time-series signal is modified based on a product of the closeness measure and the reference signal scaled by an amplification factor. The modified pixel intensity values are provided back into the source video to generate a reconstructed video such that, upon playback of the reconstructed video, viewers thereof can visually examine the amplified time-series signal, see how it is distributed and how it propagates. The methods disclosed find their uses in remote sensing applications such as telemedicine.

TECHNICAL FIELD

The present invention is directed to systems and methods for processingsource video to identify a time-series signal of interest within thatvideo and modifying pixels associated with the identified signal suchthat the signal is visually enhanced upon video playback.

BACKGROUND

In medical practice, visual enhancement of color changes appearing onskin tissue can greatly aid medical practitioners in their diagnosis ofunderlying medical conditions such as peripheral neuropathy, forexample. Methods for enhancing various aspects of images in this regardare highly desirable by diagnosticians.

Previous methods, such as that which is disclosed in “Filtering SourceVideo Data Via Independent Component Selection”, U.S. patent applicationSer. No. 13/281,975, by Mestha et al., projected the source video onto alower dimensional subspace and performed independent component analysison the projected data to identify signal components of interest whichwere then used to reconstruct the video. However, such approaches do notwork on a real-time basis since the algorithms performing IndependentComponent Analysis (ICA) are statistically-based methods which requirelarge amounts of data and are computationally intensive. Adding to theproblem is that signals of interest are often quite weak. For example,the amplitude of a cardiac pulse signal appearing on a facial region inthe RGB channels is typically less than a single unit of a 0-255 colorscale. Due to such a small signal-to-noise ratio (SNR), it can bedifficult to detect a desired signal of interest appearing in localpixels directly from local signals. On the other hand, SNR can beimproved by applying spatial filters, such as mean filters over largeareas, but it still can be difficult to identify the source of a signalin low resolution images obtained by simply averaging, as in the ICAapproach. What is desirable is to, not only enhancement of a signal ofinterest in video data, but also to ameliorate the SNR deficiency.

Accordingly, what is needed in this art is a system and method forprocessing source video to identify a time-series signal containedwithin that source video data and modify the intensity value of pixelsin the image frames of that video which are associated with that signalsuch that, upon playback, the signal is visually enhanced.

INCORPORATED REFERENCES

The following U.S. patents, U.S. patent applications, and Publicationsare incorporated herein in their entirety by reference.

“Method And Systems For Vascular Pattern Localization Using TemporalFeatures”, U.S. patent application Ser. No. 13/______, by Liu et al,discloses a method for localizing vascular pattern using temporalfeatures from video. [Attorney File No.: 20120894-US-NP]

“Removing Environment Factors From Signals Generated From Video ImagesCaptured For Biomedical Measurements”, U.S. patent application Ser. No.13/401,207, by Mestha et al. which discloses a system and method foranalyzing a video such that undesirable periodic signals and randombackground noise can be removed.

“Monitoring Respiration With A Thermal Imaging System”, U.S. patentapplication Ser. No. 13/103,406, by Xu et al., which discloses a systemand method for analyzing a video such that the respiration rate of asubject of interest in that video can be determined.

“Subcutaneous Vein Pattern Detection Via Multi-Spectral IR Imaging In AnIdentity Verification System”, U.S. patent application Ser. No.13/087,850, by Xu et al. which discloses a method for identifying anindividual in an IR video by isolating and extracting a subcutaneousvein pattern in an area of skin from that video.

“Deriving Arterial Pulse Transit Time From A Source Video Image”, U.S.patent application Ser. No. 13/401,286, by Mestha et al., whichdiscloses a method for analyzing a video such that a pulse transit timecan be determined for a person in that video.

“Estimating Cardiac Pulse Recovery From Multi-Channel Source Data ViaConstrained Source Separation”, U.S. patent application Ser. No.13/247,683 which discloses a system and method for recovering anestimated cardiac pulse rate from a sequence of RGB or multi-spectralvideo image data captured from the facial/skin region of a person oranimal being monitored for cardiac function in a remote sensingenvironment.

“Systems And Methods For Non-Contact Heart Rate Sensing”, U.S. patentapplication Ser. No. 13/247,575, by Mestha et al., which discloses asystem and method for analyzing a video of a subject of interest todetermine the subject's heart rate.

“Video-Based Estimation Of Heart Rate Variability”, U.S. patentapplication Ser. No. 13/532,057, by Mestha et al., which discloses asystem and method for estimating heart rate variability from videocaptured of a patient being monitored for cardiac function.

“Minimally Invasive Image-Based Determination Of Carbon Dioxide (CO₂)Concentration In Exhaled Breath”, U.S. patent application Ser. No.13/246,560, by Cardoso et al., which discloses a system and method foranalyzing a video such that a concentration of carbon dioxide (CO₂) in aperson's exhaled breath can be determined.

“Processing A Video For Respiration Rate Estimation”, U.S. patentapplication Ser. No. 13/529,648, by Mestha et al., which discloses asystem and method for estimating a respiration rate for a subject ofinterest captured in a video containing a view of that subject'sthoracic region.

“Processing A Video For Vascular Pattern Detection And Cardiac FunctionAnalysis”, U.S. patent application Ser. No. 13/483,992, by Mestha et al.which discloses a method for analyzing a video to identify a vascularpattern in a region of interest and then processing the pixelsassociated with the identified vascular pattern to determine variousaspects of a subject's cardiac function.

“Processing A Video For Tidal Chest Volume Estimation”, U.S. patentapplication Ser. No. 13/486,637, by Bernal et al., which discloses asystem and method for estimating tidal chest volume by analyzingdistortions in reflections of structured illumination patterns capturedin a video containing a partial view of a thoracic region of a subjectof interest being monitored for respiratory function.

“Continuous Cardiac Pulse Rate Estimation From Multi-Channel SourceVideo Data”, U.S. patent application Ser. No. 13/528,307, by Kyal etal., which discloses a system and method for continuously estimatingcardiac pulse rate from multi-channel source video captured of a patientbeing monitored for cardiac function.

“Filtering Source Video Data Via Independent Component Selection”, U.S.patent application Ser. No. 13/281,975, by Mestha et al. which disclosesa system and method for reconstructing source video data captured usinga video camera such that certain information in the source data isvisually emphasized during video playback.

“A Multi-Filter Array For A Multi-Resolution Multi-Spectral Camera”,U.S. patent application Ser. No. 13/239,642, by Xu et al., whichdiscloses a multi-filter array for a multi-resolution and multi-spectralcamera system for simultaneous spectral decomposition with a spatiallyand spectrally optimized multi-filter array suitable for image objectidentification.

“Multi-Band Infrared Camera System Optimized For Skin Detection”, U.S.patent application Ser. No. 13/416,436, by Wang et al. which discloses amethod for determining an optimal wavelength band combination forfilters of an infrared camera used for acquiring infrared imagescontaining skin tissue.

BRIEF SUMMARY

What is disclosed is a system and method for real-time enhancement of anidentified time-series signal of interest in a video that has a similarspatial and temporal structure to a given reference signal, asdetermined by a measure of closeness. A closeness measure is computedfor pixels of each image frame of each channel of a multi-channel videoto identify a time-series signal of interest. The intensity of pixelsassociated with that time-series signal is modified based on a productof the closeness measure and the reference signal scaled by anamplification factor. The modified pixel intensity values are providedback into the source video to generate a reconstructed video such that,upon playback of the reconstructed video, viewers thereof can visuallyexamine the amplified time-series signal, see how it is distributed andhow it propagates. The teachings hereof can be implemented in real-timeas image frames of video are captured on a per-channel basis. Thepresent system and method effectively enables real-time enhancement ofan identified time-series signal of interest during video acquisition.The methods disclosed find their uses in a wide array of remote sensingapplications including the telemedicine arts.

One embodiment of the present method for real-time enhancement of asource video signal involves the following. First, source video isacquired by a multi-channel video acquisition system over multiplechannels. For each pixel location (i, j) of each channel k of themulti-channel source video signal, a measure of closeness is calculatedand used, in conjunction with a reference signal R(t), to identify atime-series signal I(t) of interest where I_(ijk)(t) is the signal thatappears at pixel location (i, j, k) in the video. Embodiments of acloseness measure are disclosed. Intensity of pixels associated with theidentified time-series signal is modified, in a manner disclosed hereinin detail, to produce modified video data. The modified video data isretrofitted back into the source video to generate a reconstructedvideo. The reconstructed video is communicated, in real-time, to adisplay device for viewing.

Many features and advantages of the above-described method will becomereadily apparent from the following detailed description andaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features and advantages of the subject matterdisclosed herein will be made apparent from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a flow diagram of one example embodiment of the present methodfor real-time enhancement and visualization of source video;

FIG. 2 is a continuation of the flow diagram of FIG. 1 with flowprocessing continuing with respect to either node A or B;

FIG. 3 illustrates a block diagram of one embodiment of a videoprocessing system for performing real-time enhancement of a signal ofinterest in a source video as described with respect to the flowdiagrams of FIGS. 1 and 2; and

FIG. 4 illustrates a block diagram of one example special purposecomputer for implementing one or more aspects of the present method asdescribed with respect to the flow diagrams of FIGS. 1 and 2 and theblock diagram of FIG. 3.

DETAILED DESCRIPTION

What is disclosed is a system and method for processing source video toidentify a time-series signal contained within that source video dataand modify the intensity value of pixels in the image frames of thatvideo associated with that signal such that, upon playback, the signalis visually enhanced. The teachings disclosed herein enable users to seethe enhanced signal, and see how it is distributed and how itpropagates.

NON-LIMITING DEFINITIONS

“Source video” refers to a time varying video signal acquired over Cchannels where k=1 . . . C, using a multi-channel video acquisitionsystem as defined herein. The acquired video signal comprises aplurality of image frames. Each frame of the video is a 2D array ofpixels with each pixel location (i, j, k) in the video having anintensity I_(ijk) that corresponds to an amount of detected reflectedenergy projected by an illumination source over a wavelength range ofinterest.

“Receiving source video” is intended to be widely construed and means toretrieve, capture with a video camera, acquire, or otherwise obtain asource video for processing in accordance with the present method. Thesource video can be retrieved from a memory or storage device which maybe internal to the video acquisition system, or obtained from a remotedevice over a network or from a website. The video data may also beretrieved from a removable media such as, for example, a CDROM, DVD, USBDrive or a memory card.

A “multi-channel video acquisition system” is a device for acquiring asource video signal V^(H×W) over C channels where each image frame has asize H×W, where H is the height of an image frame in pixels and W is thewidth of the image frame in pixels. In one embodiment, the multi-channelvideo acquisition system is a device with a high frame rate and highspatial resolution such as, for example, a monochrome video camera forcapturing black/white video images, or a color video camera forcapturing color video images. The multi-channel video acquisition systemmay comprise a multi-spectral or hyperspectral video camera. Suchspectral sensing device have relatively low frame rates and low spatialresolution but high spectral resolution. The video system may be ahybrid device capable of operating in dual modes, i.e., a conventionalvideo mode with high frame rate and high spatial resolution, and aspectral mode with high spectral resolution. Multi-channel videoacquisition systems comprising standard video cameras and those deviceswith spectral sensors are readily available from different vendors invarious streams of commerce. Multi-channel video devices typically havea plurality of outputs from which the acquired source video can bereceived on a per-channel basis. Video systems may further comprise oneor more processors capable of executing machine readable programinstructions and may further comprise a video analysis module foranalyzing and reconstructing the source video in real-time, inaccordance with the teachings hereof.

A “video analysis module”, in one embodiment, comprises at least oneprocessor executing machine readable program instructions for analyzingvideo images such that the video can be reconstructed and communicatedto a display device. A video analysis module may comprise, in whole orin part, a software application working alone or in conjunction withhardware resources such as, for example, a dedicated processor or ASIC.Aspects of a video analysis module may leverage off-the-shelf software.

A “time-series signal of interest” is a signal I(t) within the sourcevideo which contains meaningful data which is desired to be visuallyenhanced using the methods disclosed herein. Signal I_(ijk)(t) appearsat pixel location (i, j, k) in the video. A signal of interest may be,for example, a subject's cardiac pulse, their respiration rate, bloodflow, CO₂ concentration, or perspiration rate. The signal of interestmay be a detectable brainwave signal and may further include physicalmotion such as eye blinks, twitches, ticks, and the like. Thetime-series signal of interest is identified in the source video using ameasure of closeness determined with respect to a reference signal inboth a spatial and time direction.

A “reference signal” is a signal used to facilitate the identificationof a time-series signal of interest in the source video. For example, inthose embodiments where the subject's cardiac pulse frequency is thetime-series signal of interest intended to be enhanced in the sourcevideo, the reference signal R(t) would be a signal which has a frequencyrange which approximates the frequency range of the subject's cardiacpulse frequency. A reference signal can be an EKG signal, an ECG signal,or a photoplethysmographic (PPG) signal obtained using non-invasiveelectro-optic techniques which sense cardiovascular pulse waves (alsoreferred to as “blood volume pulse”) through variations in transmittedor reflected light. EKG, ECG, PPG signals and the like, provide valuableinformation about various autonomic functions. The reference signal canbe any signal acquired or generated using any of a variety of systems,devices, methods, or techniques such as, for instance, a Blind SourceSeparation (BSS) technique which recovers unobserved source signals froma set of observed mixed signals without any prior information beingknown about the “mixing” process. The reference signal may be modifiedwith respect to a frequency, amplitude, phase or waveform as needed,depending on the implementation and the nature of the time-series signalof interest intended to be identified in the source video forenhancement. The reference signal may be received in real-time from asensing device capable of generating a reference signal as output. Thereference signal may be derived, generated or otherwise obtainedbeforehand, stored to a memory or storage device, and subsequentlyretrieved when needed. The reference signal facilitates theidentification of a time-series signal of interest in the source videousing a measure of closeness.

A “measure of closeness”, or “closeness measure” is calculated for eachpixel (i, j) of each channel k of the source video and used to determinehow close a given signal at a pixel location is with respect to areference signal. The closeness measure effectively identifies thetime-series signal of interest in the source video. One embodiment ofthe measure of closeness is defined by the following relationship:

C _(ijk)(t+1)=βC _(ijk)(t)+(I _(ijk)(t)−Ī _(ijk)(t))(R(t)− R (t))

Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),

R (t+1)=α R (t)+(1−α)R(t),  (1)

where 0<β<1, 0<α<1. In Eq. (1), Ī_(ijk)(t) and R(t) approximate movingaverages of video signals I_(ijk)(t) and reference signal R(t). Aftermean-removal, the empirical correlation between I_(ijk)(t)−Ī_(ijk)(t)and R(t)− R(t) is calculated (as shown in Eq. (1)) to evaluate thedegree of “similarity” between these signals. In various embodiments,this is done in an online fashion.

In those embodiments where the video signal is expected to contain asimilar feature to the reference signal with time lags, one can preparea set of candidate time lags τ_(l), where l=1, 2, . . . , m, and m is auser-specified number of shifts. For each time lag τ_(l), the measure ofcloseness between video signal I_(ijk)(t) and the time-shifted referencesignal R(t−τ_(l)) is given by:

C _(ijk) ^(τ) ^(l) (t+1)=βC _(ijk) ^(τ) ^(l) (t)+(I _(ijk)(t)−Ī_(ijk)(t))(R(t−τ _(l))− R (t−τ _(l)))

Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),

R (t+1)=α R (t)+(1−α)R(t),  (2)

“Modifying video data” means modifying intensity values of pixels in thesource video which are associated with the identified time-series signalof interest. In those embodiments where the measure of closeness isdefined by the relationship of Eq. (1), the modification applied topixel location (i, j, k) which is associated with identified signalI_(ijk)(t) is given by:

J _(ijk)(t)=I _(ijk)(t)+δC _(ijk)(t)R(t).  (3)

where δ is a user-defined amplification factor.

In the embodiment of the closeness measure of Eq. (2) where thereference signal has been shifted by a pre-defined shift τ_(l) themodification applied to pixel location (i, j, k) is given by:

$\begin{matrix}{{J_{ijk}(t)} = {{I_{ijk}(t)} + {\delta {\sum\limits_{l = 1}^{m}{C_{ijk}{\,^{\,^{\tau}l}(t)}{{R\left( {t - \tau_{l}} \right)}.}}}}}} & (4)\end{matrix}$

The introduction of multiple lags in Eq. (2) and Eq. (4) is intended toreveal spatio-temporal structure of the signal of interest. While thevisual effect obtained by (3) only captures and amplifies the signal ofinterest appearing in the video without any time lag, (4) also capturesand amplifies the signal of interest appearing in the video with sometime lags. This visualizes dynamic propagation of the featured signaland allows more vivid video reconstruction.

“Retrofitting” the modified video data back into the source video meansto replace the pixel value at location (i, j, k) with the valueJ_(ijk)(t). Such a replacement, performed on a pixel by pixel, channelby channel basis, effectively generates a reconstructed source video.

A “reconstructed source video” means intensity values of pixelsassociated with the identified time-series signal of interest in thesource video have been replaced with the modified video data such that,upon playback, certain pixel locations in each image frame of eachchannel of the multi-channel source video have been visually enhanced.The reconstructed source video is communicated to a display device.

“Communicating the reconstructed video” is intended to be widelyconstrued and means to send, transmit, deliver, otherwise provide thereconstructed source video to a display device.

A “display device” refers to any device capable of receiving thereconstructed source video and storing that video for eventual display.Display devices, as are generally understood, include LCD, HD, CRT, andtouchscreen displays. Such devices have specialized video processors andcircuitry which operate in conjunction with specialized memory such as,for instance, raster memory. A size of the raster memory depends on theresolution of the display device. When the image to be visuallydisplayed is stored in the raster memory, the specialized videoprocessing circuitry reads the data stored in those raster memorylocations for visual presentation.

Flow Diagram of One Embodiment

Reference is now being made to the flow diagram of FIG. 1 whichillustrates a flow diagram of one example embodiment of the presentmethod for real-time enhancement of a signal of interest in a video.Flow processing begins at 100 and immediately proceeds to step 102.

At step 102, receive a source video for processing. The received sourcevideo contains a time-series signal of interest which is intended to beidentified and enhanced. The source video comprises a plurality of imageframes with each frame comprising an array of pixels. Each pixel has anintensity value corresponding to detected reflected energy projected byan illumination source over a wavelength range of interest. The sourcevideo is acquired using a multi-channel video acquisition system with Cchannels, where channel index k=1 . . . C.

At step 104, select a first channel k of the multi-channel source videofor reconstruction. Such a selection can be effectuated using, forexample, a graphical user interface such as that which is shown anddescribed with respect to the embodiment of FIG. 3. The channels can beindividually selected for processing or processed according to a pre-setor pre-programmed sequential method.

At step 106, select a first pixel location (i, j, k) for processing.Pixel of each image frame of the selected channel of the source videocan be processed using, for instance, a row/column iterative selectionmethod which steps through each pixel location of each image frame ofthe channel selected at step 104. Example image frames are shown at 302of FIG. 3.

In some embodiments, the pixel location (i, j, k) can be selected forprocessing only along the vascular pattern. Vascular patterns may beidentified using techniques disclosed in

At step 108, calculate, for the selected pixel location, a measure ofcloseness with respect to a reference signal R(t). Embodiments fordifferent closeness measures are described with respect to Eqs. (1) and(2). Such a calculation is performed by a processor executing machinereadable program instructions for calculating a closeness measure for agiven pixel location.

At step 110, use the measure of closeness to identify a time-seriessignal I(t) of interest in the received source video. If the closenessmeasure is within a pre-defined threshold then the signal at thatlocation is determined to be “close” enough to qualify for enhancement.

At step 112, a determination is made whether a time-series signal ofinterest has been identified at this pixel location (i, j, k). If so,then processing continues with respect to node A of FIG. 2. If not, thenprocessing continues with respect node B.

Reference is now being made to the flow diagram of FIG. 2 which is acontinuation of the flow diagram of FIG. 1 with flow processingcontinuing with respect to either node A or node B.

At step 114, modify an intensity of the pixel at this location togenerate modified video data. Embodiments for J_(ijk) (t) are describedin Eqs. (3) and (4), depending on the closeness measure used in step108. The modified video data may be stored to a memory or storage devicefor subsequent retrieval.

At step 116, retrofit the modified video data into the source video toobtain a reconstructed source video. Retrofitting effectively replacesthe intensity value of the pixel at this location in the source videowith value of the modified video data (of step 114).

At step 118, communicate the reconstructed source video to a displaydevice. Thereafter, processing continues with respect to step 120.

If, as a result of the determination of step 112 of FIG. 1, thetime-series signal of interest has not been identified at this pixellocation the processing continues with respect to node B of FIG. 2wherein, at step 120, a determination is made whether any more pixelsremain to be processed. If so, then processing continues with respect tonode C of FIG. 1 wherein, at step 106, a next pixel location is selectedfor processing. Processing then repeats in a similar manner with respectto the next selected pixel location until all desired pixels have beenselected for processing. If, at step 120, no more pixels remain to beprocessed in any of the image frames of this channel of source videothen processing continues with respect to step 122 wherein adetermination is made whether any more channels of source video remainto be selected. If more channels of video data remain to be processedthen processing repeats with respect to node D wherein, at step 104wherein a next channel of source video is selected or otherwiseidentified for processing. Processing repeats for all pixel locations ofall image frames for this next selected channel until all desiredchannels of the multi-channel source video have been processed.Thereafter, in this embodiment, further processing stops.

In the embodiment of FIGS. 1 and 2, the step of communicating isperformed as the modified video data is retrofitted into the sourcevideo as a result of the determination of step 112. It should beappreciated that the step of communicating may be performed at variousprocessing intervals with the reconstructed video being stored to amemory or storage device, at least temporarily, and subsequentlyretrieved as needed or as the available amount of memory or storagefalls below a defined limit. In other embodiments, the reconstructedsource video is communicated upon completion of processing all the pixellocations of a given image frame of source video, or after havingprocessed all the frames of a given channel. In yet other embodiments,the modified video data is stored and retrieved and retrofitted into thesource video to generate the reconstructed video as that data begins toaccumulate in memory, and the reconstructed video thereafter iscommunicated to the display device. Such embodiments are intended tofall within the scope of the appended claims. Various aspects of themethods disclosed herein are intended to be performed with a degree ofparallelism with respect to other steps hereof.

It should also be appreciated that the flow diagrams hereof areillustrative. One or more of the operative steps may be performed in adiffering order. Other operations, for example, may be added, modified,enhanced, integrated, or consolidated. Such variations are intended tofall within the scope of the appended claims. All or portions of theflow diagrams may be implemented partially or fully in hardwareoperating in conjunction software.

Block Diagram Of Video Processing System

Reference is now being made to FIG. 3 which illustrates a block diagramof one embodiment of a system for performing real-time enhancement of asignal of interest in a source video as described with respect to theflow diagrams of FIGS. 1 and 2.

In FIG. 3, the source video, shown as a plurality of image frames 302,is captured using a multi-channel video acquisition system 303. Theacquired source video 302 is provided to video processing system 304which, in this embodiment, is shown comprising a plurality of modulesand processing units. Closeness Measure Processor 305 receives theacquired source video 302, and determines, on a per-frame per-channelbasis, a measure of closeness with respect to a reference signal R(t)obtained from Reference Signal Generator 306 such that a time-seriessignal of interest I(t) can be identified. The reference signal andvarious mathematical formulas and variables, and the like, are retrievedon an as-needed basis from storage device 307. Once the time-seriessignal of interest has been identified, Intensity Modifier Module 308modifies the intensity values of pixels associated with the identifiedsignal to generate, as output, modified video data. The modified videodata is stored to storage device 307. Moreover, various mathematicalformulas and variables are retrieved by Module 308 from storage device307 to perform its intended function. Module 308 further retrofits themodified video data into the source video to generate a reconstructedsource video. Processor 309, which has its own memory 310, is utilizedby any of the modules of the video processing system 304 to facilitateprocessing and communication between the various modules and withstorage device 307. Processor 309 communicates (at 311) thereconstructed source video to workstation 312 for display. Theworkstation is shown generally comprising a display device 313 fordisplaying the reconstructed source video. The workstation furthereffectuates a user input or selection such as, for example, the aselection as to a channel of video or pixel locations of image frames toprocess. Display device 313 may be placed in communication with any ofthe modules and processors of the video processing system 304 and/or themulti-channel video acquisition system 303 such that images processedthereby can be viewed in real-time. A user or technician of the systemof FIG. 3 may use the graphical user interface of workstation 312, e.g.,keyboard 314 and mouse (not shown), to set parameters enter values,select pixels, channels, and/or regions of images for processing, and todetermine which device is to receive the communicated reconstructedsource video. In one embodiment, the user uses a slideably selectablebar to set the amplification factor δ such that the amount of visualenhancement is made adjustable in real-time. Values entered andselections made may be stored to storage medium 315 or to computerreadable media 316. It should be appreciated that some or all of thefunctionality performed by any of the modules or processing units of thevideo processing system of FIG. Information stored to a computerreadable media can be retrieved by a media reader such as, for example,a CD-ROM or DVD drive. Workstation 312 is shown having been placed incommunication with network 317 via a communications interface (notshown).

It should be appreciated that the workstation of FIG. 3 has an operatingsystem and other specialized software configured to display a variety ofnumeric values, text, scroll bars, pull-down menus with user selectableoptions, and the like, for entering, selecting, or modifying informationdisplayed thereon. Any of the modules and processing units of FIG. 3 canbe placed in communication with display device 313 and database 315 andmay store/retrieve therefrom data, variables, records, parameters,functions, mathematical representations, and any machine readableprogram instructions as required to perform their intended functions.Any of the modules and processing units of system of FIG. 3 may beplaced in communication with one or more devices over network 317.

Various modules may designate one or more components which may, in turn,comprise software and/or hardware designed to perform their intendedfunctions. A plurality of modules may collectively perform a singlefunction. Each module may have a specialized processor therein forexecuting machine readable program instructions. A module may comprise asingle piece of hardware such as an ASIC, electronic circuit, or specialpurpose computer such as that which is shown and discussed with respectto FIG. 4. A plurality of modules may be executed by either a singlespecial purpose computer or a plurality of special purpose computersystems operating in parallel. Connections between modules include bothphysical and logical connections. Modules may further include one ormore software/hardware modules which may comprise an operating system,drivers, device controllers, and other apparatuses some or all of whichmay be connected via a network.

Example Special Purpose Computer

Reference is now being made to FIG. 4 which illustrates a block diagramof one example special purpose computer 400 for implementing one or moreaspects of the present method as described with respect to the flowdiagrams of FIGS. 1 and 2 and the block diagram of FIG. 3. Such aspecial purpose processor is capable of executing machine executableprogram instructions and may comprise any of a micro-processor,micro-controller, ASIC, electronic circuit, or any combination thereof.

In FIG. 4, communications bus 402 is in communication with a centralprocessing unit (CPU) 404 capable of executing machine readable programinstructions for performing any of the calculations, comparisons,logical operations, and other program instructions for performing any ofthe steps described above with respect to the flow diagrams andillustrated embodiments hereof. Processor 404 is in communication withmemory (ROM) 406 and memory (RAM) 408 which collectively constituteexample storage devices. Such memory may be used to store machinereadable program instructions and other program data and results tosufficient to carry out any of the functionality described herein. Diskcontroller 410 interfaces with one or more storage devices 414 which maycomprise external memory, zip drives, flash memory, USB drives, or otherdevices such as CD-ROM drive 412 and floppy drive 416. Storage devicestores machine executable program instructions for executing the methodshereof. Such storage devices may be used to implement a database whereinvarious records are stored. Display interface 418 effectuates thedisplay of information on display device 420 in various formats such as,for instance, audio, graphic, text, and the like. Interface 424effectuates a communication via keyboard 426 and mouse 428, collectivelya graphical user interface. Such a graphical user interface is usefulfor a user to enter information about any of the displayed informationin accordance with various embodiments hereof. Communication withexternal devices may occur using example communication port(s) 422. Onesuch external device placed in communication with the special purposecomputer system of FIG. 4 is the spectral multi-channel video camera 303of FIG. 3. Such ports may be placed in communication with any of themodules and components of the example networked configuration of FIG. 3,as shown and described herein, using the Internet or an intranet, eitherby direct (wired) link or wireless link. Example communication portsinclude modems, network cards such as an Ethernet card, routers, aPCMCIA slot and card, USB ports, and the like, capable of transferringdata from one device to another. Software and data is transferred viathe communication ports in the form of signals which may be any ofdigital, analog, electromagnetic, optical, infrared, or other signalscapable of being transmitted and/or received by the communicationsinterface. Such signals may be implemented using, for example, a wire,cable, fiber optic, phone line, cellular link, RF, or other signaltransmission means presently known in the arts or which have beensubsequently developed.

It will be appreciated that the above-disclosed and other features andfunctions, or alternatives thereof, may be desirably combined into manyother different systems or applications. Various presently unforeseen orunanticipated alternatives, modifications, variations, or improvementstherein may become apparent and/or subsequently made by those skilled inthe art which are also intended to be encompassed by the followingclaims. Accordingly, the embodiments set forth above are considered tobe illustrative and not limiting. Various changes to the above-describedembodiments may be made without departing from the spirit and scope ofthe invention.

The teachings hereof can be implemented in hardware or software usingany known or later developed systems, structures, devices, and/orsoftware by those skilled in the applicable art without undueexperimentation from the functional description provided herein with ageneral knowledge of the relevant arts. Moreover, the methods hereof canbe implemented as a routine embedded on a personal computer or as aresource residing on a server or workstation, such as a routine embeddedin a plug-in, a driver, or the like. The teachings hereof may bepartially or fully implemented in software using object orobject-oriented software development environments that provide portablesource code that can be used on a variety of computer, workstation,server, network, or other hardware platforms. One or more of thecapabilities hereof can be emulated in a virtual environment as providedby an operating system, specialized programs or leverage off-the-shelfcomputer graphics software such as that in Windows, Java, or from aserver or hardware accelerator or other image processing devices.

One or more aspects of the methods described herein are intended to beincorporated in an article of manufacture, including one or morecomputer program products, having computer usable or machine readablemedia. The article of manufacture may be included on a storage devicereadable by a machine architecture embodying executable programinstructions capable of performing the methodologies described herein.The article of manufacture may be included as part of a standalonesystem, an operating system, or a software package which may be shipped,sold, leased, or otherwise provided either alone or as part of anadd-on, update, upgrade, or product suite. It will be appreciated thatvarious features and functions and alternatives hereof may be combinedinto other systems or applications which are heretofore unknown.

Various presently unforeseen or unanticipated alternatives,modifications, variations, or improvements therein may become apparentand/or subsequently made by those skilled in the art which are alsointended to be encompassed by the following claims. Accordingly, theembodiments set forth above are considered to be illustrative and notlimiting. Changes to the above-described embodiments may be made withoutdeparting from the spirit and scope of the invention. The teachings ofany printed publications including patents and patent applications, areeach separately hereby incorporated by reference in their entirety.

What is claimed is:
 1. A method for real-time enhancement of a signal ofinterest in a video, the method comprising: receiving a source videoacquired using a multi-channel video acquisition system with C channelswhere channel k=1 . . . C, said source video comprising a plurality ofimage frames, each frame comprising an array of pixels, each pixelhaving an intensity value corresponding to detected reflected energyprojected by an illumination source over a wavelength range of interest;for each pixel location (i, j) of each image frame of each channel k ofsaid source video: determining a measure of closeness with respect to areference signal R(t) in both a spatial and time direction; identifying,using said measure of closeness, a time-series signal I(t) of interestin said source video, where signal I_(ijk)(t) appears at pixel location(i, j, k); in response to having identified said time-series signal ofinterest at pixel location (i, j, k): modifying an intensity of saidpixel at said location to generate modified video data J_(ijk) (t); andretrofitting said modified video data into said source video to generatea reconstructed source video; and communicating said reconstructedsource video to a display device.
 2. The method of claim 1, wherein saidmulti-channel video acquisition system comprises any combination of: amonochrome video camera, a color video camera, a multi-spectral camera,a hyper-spectral camera, and a hybrid device.
 3. The method of claim 1,wherein said measure of closeness at pixel location (i, j, k) comprises:C _(ijk)(t+1)=βC _(ijk)(t)+(I _(ijk)(t)−Ī _(ijk)(t))(R(t)− R (t)),Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),R (t+1)=α R (t)+(1−α)R(t), where 0<β<1, and 0<α<1.
 4. The method ofclaim 3, wherein said modification at pixel location (i, j, k)comprises:J _(ijk)(t)=I _(ijk)(t)+δC _(ijk)(t)R(t).
 5. The method of claim 1,further comprising shifting said reference signal by a pre-specifiedshift τ_(l), where l=1, 2, . . . , m, and m is a user-specified numberof shifts.
 6. The method of claim 5, wherein said measure of closenessat pixel location (i, j, k) comprises:C _(ijk) ^(τ) ^(l) (t+1)=βC _(ijk) ^(τ) ^(l) (t)+(I _(ijk)(t)−Ī_(ijk)(t))(R(t−τ _(l))− R (t−τ _(l))),Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),R (t+1)=α R (t)+(1−α)R(t), where 0<β<1, and 0<α<1.
 7. The method ofclaim 6, wherein said modification at pixel location (i, j, k)comprises:${J_{ijk}(t)} = {{I_{ijk}(t)} + {\delta {\sum\limits_{l = 1}^{m}{C_{ijk}{\,^{\,^{\tau}l}(t)}{{R\left( {t - \tau_{l}} \right)}.}}}}}$8. The method of claim 1, wherein said reference signal is extractedfrom said source video.
 9. The method of claim 1, wherein saidtime-series signal is a signal of a subject's cardiac pulse, and whereinsaid reference signal is a signal which has a frequency range thatapproximates a frequency range of said subject's cardiac pulse.
 10. Themethod of claim 1, further comprising using said reconstructed sourcevideo to facilitate a determination of any of: a cardiac signal,respiratory rate and function, CO₂ concentration in exhaled breath,perspiration, brainwave pattern and signals, degree of infection, bloodflow and circulation, and a bodily motion.
 11. The method of claim 1,wherein said pixel location (i, j) of each image frame of each channel kof said source video is along a vascular pattern.
 12. A system forreal-time enhancement of a signal of interest in a video, the systemcomprising: a multi-channel video acquisition system for acquiringsource video with C channels, where channel k=1 . . . C; and a processorin communication with a memory and said multi-channel video acquisitionsystem, said processor executing machine readable instructions forperforming: receiving a source video acquired using said multi-channelvideo acquisition system, said source video comprising a plurality ofimage frames, each frame comprising an array of pixels, each pixelhaving an intensity value corresponding to detected reflected energyprojected by an illumination source over a wavelength range of interest;for each pixel location (i, j) of each image frame of each channel k ofsaid source video: determining a measure of closeness with respect to areference signal R(t) in both a spatial and time direction; identifying,using said measure of closeness, a time-series signal I(t) of interestin said source video, where signal I_(ijk)(t) appears at pixel location(i, j, k); and in response to having identified said time-series signalof interest at pixel location (i, j, k): modifying an intensity of saidpixel at said location to generate modified video data J_(ijk)(t); andretrofitting said modified video data into said source video to generatea reconstructed source video; and communicating said reconstructedsource video to a display device.
 13. The system of claim 12, whereinsaid multi-channel video acquisition system comprises any combinationof: a monochrome video camera, a color video camera, a multi-spectralcamera, a hyper-spectral camera, and a hybrid device.
 14. The system ofclaim 12, wherein said measure of closeness at pixel location (i, j, k)comprises:C _(ijk)(t+1)=βC _(ijk)(t)+(I _(ijk)(t)−Ī _(ijk)(t))(R(t)− R (t)),Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),R (t+1)=α R (t)+(1−α)R(t), where 0<β<1, and 0<α<1.
 15. The system ofclaim 14, wherein said modification at pixel location (i, j, k)comprises:J _(ijk)(t)=I _(ijk)(t)+δC _(ijk)(t)R(t).
 16. The system of claim 12,further comprising shifting said reference signal by a pre-specifiedshift τ_(l), where l=1, 2, . . . , m, and m is a user-specified numberof shifts.
 17. The system of claim 16, wherein said measure of closenessat pixel location (i, j, k) comprises:C _(ijk) ^(τ) ^(l) (t+1)=βC _(ijk) ^(τ) ^(l) (t)+(I _(ijk)(t)−Ī_(ijk)(t))(R(t−τ _(l))− R (t−τ _(l))),Ī _(ijk)(t+1)=αĪ _(ijk)(t)+(1−α)I _(ijk)(t),R (t+1)=α R (t)+(1−α)R(t), where 0<β<1, and 0<α<1.
 18. The system ofclaim 17, wherein said modification at pixel location (i, j, k)comprises:${J_{ijk}(t)} = {{I_{ijk}(t)} + {\delta {\sum\limits_{l = 1}^{m}{C_{ijk}{\,^{\,^{\tau}l}(t)}{{R\left( {t - \tau_{l}} \right)}.}}}}}$19. The system of claim 12, wherein said reference signal is extractedfrom said source video.
 20. The system of claim 12, wherein saidtime-series signal is a signal of a subject's cardiac pulse, and whereinsaid reference signal is a signal which has a frequency range thatapproximates a frequency range of said subject's cardiac pulse.
 21. Thesystem of claim 12, further comprising using said reconstructed sourcevideo to facilitate a determination of any of: a cardiac signal,respiratory rate and function, CO₂ concentration in exhaled breath,perspiration, brainwave pattern and signals, degree of infection, bloodflow and circulation, and a bodily motion.
 22. The system of claim 12,wherein said pixel location (i, j) of each image frame of each channel kof said source video is along a vascular pattern.